We thank the reviewers for their insights and suggestions. Answers below will be included in expanded discussions in future versions of the paper. In the case of R3's car example, as long as states from 10 steps into the future are sampled This is discussed in L211-L215 in Section 6 "Practical Training of γ -Models". The only Monte Carlo trajectory estimates are in the final column for comparison.

We thank all our reviewers for their feedback! We will respond to (R2, R3) separately to R1 due to different concerns. We thank R2 and R3 for their vote of confidence and giving this work at high score of 9 and 8 respectively. It means a lot to us - to see our ideas accepted by our peers at NeurIPS who also believe that our "work opens many new We experimented with setting all weights to a single fixed value, e.g. However, if we then nudge that value by a small amount, to say 0.6, the network fails completely at the In fact, the best performing values were outside of this training set. We will cite and discuss this work in our revised paper. NeurIPS2019 will discuss similar themes and we are excited to see more ideas in this direction from both communities. We agree with R3 that scaling up is the next step. Stanley2009) to scale W ANNs architectures to scales able to compete on benchmarks such as ImageNET and Atari. We wish to take the time to conduct this investigation thoroughly, and plan to report the findings in a follow up paper. W ANNs. We would also like to thank R3 for the other minor suggestions, we will clarify the labels and information. In the spirit of this extreme experiment the algorithm used was purposefully kept simple. Our original intention was to focus only on continuous-control RL experiments, and decided to run MNIST "for fun" We could have confined the paper to only RL experiments (most RL papers don't run MNIST Finally, we do believe there is a connection to the neuroscience field. "What Artificial Neural Networks can Learn from Animal Brains" (Zador2019) whose central theme is that "The first